92 research outputs found
Using Genetic Programming to Build Self-Adaptivity into Software-Defined Networks
Self-adaptation solutions need to periodically monitor, reason about, and
adapt a running system. The adaptation step involves generating an adaptation
strategy and applying it to the running system whenever an anomaly arises. In
this article, we argue that, rather than generating individual adaptation
strategies, the goal should be to adapt the control logic of the running system
in such a way that the system itself would learn how to steer clear of future
anomalies, without triggering self-adaptation too frequently. While the need
for adaptation is never eliminated, especially noting the uncertain and
evolving environment of complex systems, reducing the frequency of adaptation
interventions is advantageous for various reasons, e.g., to increase
performance and to make a running system more robust. We instantiate and
empirically examine the above idea for software-defined networking -- a key
enabling technology for modern data centres and Internet of Things
applications. Using genetic programming,(GP), we propose a self-adaptation
solution that continuously learns and updates the control constructs in the
data-forwarding logic of a software-defined network. Our evaluation, performed
using open-source synthetic and industrial data, indicates that, compared to a
baseline adaptation technique that attempts to generate individual adaptations,
our GP-based approach is more effective in resolving network congestion, and
further, reduces the frequency of adaptation interventions over time. In
addition, we show that, for networks with the same topology, reusing over
larger networks the knowledge that is learned on smaller networks leads to
significant improvements in the performance of our GP-based adaptation
approach. Finally, we compare our approach against a standard data-forwarding
algorithm from the network literature, demonstrating that our approach
significantly reduces packet loss.Comment: arXiv admin note: text overlap with arXiv:2205.0435
Evaluating Model Testing and Model Checking for Finding Requirements Violations in Simulink Models
Matlab/Simulink is a development and simulation language that is widely used
by the Cyber-Physical System (CPS) industry to model dynamical systems. There
are two mainstream approaches to verify CPS Simulink models: model testing that
attempts to identify failures in models by executing them for a number of
sampled test inputs, and model checking that attempts to exhaustively check the
correctness of models against some given formal properties. In this paper, we
present an industrial Simulink model benchmark, provide a categorization of
different model types in the benchmark, describe the recurring logical patterns
in the model requirements, and discuss the results of applying model checking
and model testing approaches to identify requirements violations in the
benchmarked models. Based on the results, we discuss the strengths and
weaknesses of model testing and model checking. Our results further suggest
that model checking and model testing are complementary and by combining them,
we can significantly enhance the capabilities of each of these approaches
individually. We conclude by providing guidelines as to how the two approaches
can be best applied together.Comment: 10 pages + 2 page reference
Approximation-Refinement Testing of Compute-Intensive Cyber-Physical Models: An Approach Based on System Identification
Black-box testing has been extensively applied to test models of Cyber-Physical systems (CPS) since these models are not often amenable to static and symbolic testing and verification. Black-box testing, however, requires to execute the model under test for a large number of candidate test inputs. This poses a challenge for a large and practically-important category of CPS models, known as compute-intensive CPS (CI-CPS) models, where a single simulation may take hours to complete. We propose a novel approach, namely ARIsTEO, to enable effective and efficient testing of CI-CPS models. Our approach embeds black-box testing into an iterative approximation-refinement loop. At the start, some sampled inputs and outputs of the CI-CPS model under test are used to generate a surrogate model that is faster to execute and can be subjected to black-box testing. Any failure-revealing test identified for the surrogate model is checked on the original model. If spurious, the test results are used to refine the surrogate model to be tested again. Otherwise, the test reveals a valid failure. We evaluated ARIsTEO by comparing it with S-Taliro, an open-source and industry-strength tool for testing CPS models. Our results, obtained based on five publicly-available CPS models, show that, on average, ARIsTEO is able to find 24% more requirements violations than S-Taliro and is 31% faster than S-Taliro in finding those violations. We further assessed the effectiveness and efficiency of ARIsTEO on a large industrial case study from the satellite domain. In contrast to S-Taliro, ARIsTEO successfully tested two different versions of this model and could identify three requirements violations, requiring four hours, on average, for each violation
Learning Non-robustness using Simulation-based Testing: a Network Traffic-shaping Case Study
An input to a system reveals a non-robust behaviour when, by making a small
change in the input, the output of the system changes from acceptable (passing)
to unacceptable (failing) or vice versa. Identifying inputs that lead to
non-robust behaviours is important for many types of systems, e.g.,
cyber-physical and network systems, whose inputs are prone to perturbations. In
this paper, we propose an approach that combines simulation-based testing with
regression tree models to generate value ranges for inputs in response to which
a system is likely to exhibit non-robust behaviours. We apply our approach to a
network traffic-shaping system (NTSS) -- a novel case study from the network
domain. In this case study, developed and conducted in collaboration with a
network solutions provider, RabbitRun Technologies, input ranges that lead to
non-robustness are of interest as a way to identify and mitigate network
quality-of-service issues. We demonstrate that our approach accurately
characterizes non-robust test inputs of NTSS by achieving a precision of 84%
and a recall of 100%, significantly outperforming a standard baseline. In
addition, we show that there is no statistically significant difference between
the results obtained from our simulated testbed and a hardware testbed with
identical configurations. Finally we describe lessons learned from our
industrial collaboration, offering insights about how simulation helps discover
unknown and undocumented behaviours as well as a new perspective on using
non-robustness as a measure for system re-configuration.Comment: This paper is accepted at the 16th IEEE International Conference on
Software Testing, Verification and Validation (ICST 2023
Automated Test Suite Generation for Time-Continuous Simulink Models
All engineering disciplines are founded and rely on models, al- though they may differ on purposes and usages of modeling. Inter- disciplinary domains such as Cyber Physical Systems (CPSs) seek approaches that incorporate different modeling needs and usages. Specifically, the Simulink modeling platform greatly appeals to CPS engineers due to its seamless support for simulation and code generation. In this paper, we propose a test generation approach that is applicable to Simulink models built for both purposes of simulation and code generation. We define test inputs and outputs as signals that capture evolution of values over time. Our test gener- ation approach is implemented as a meta-heuristic search algorithm and is guided to produce test outputs with diverse shapes according to our proposed notion of diversity. Our evaluation, performed on industrial and public domain models, demonstrates that: (1) In con- trast to the existing tools for testing Simulink models that are only applicable to a subset of code generation models, our approach is applicable to both code generation and simulation Simulink mod- els. (2) Our new notion of diversity for output signals outperforms random baseline testing and an existing notion of signal diversity in revealing faults in Simulink models. (3) The fault revealing ability of our test generation approach outperforms that of the Simulink Design Verifier, the only testing toolbox for Simulink
Automated Repair of Feature Interaction Failures in Automated Driving Systems
In the past years, several automated repair strategies have been
proposed to fix bugs in individual software programs without any
human intervention. There has been, however, little work on how
automated repair techniques can resolve failures that arise at the
system-level and are caused by undesired interactions among different
system components or functions. Feature interaction failures
are common in complex systems such as autonomous cars that are
typically built as a composition of independent features (i.e., units
of functionality). In this paper, we propose a repair technique to
automatically resolve undesired feature interaction failures in automated
driving systems (ADS) that lead to the violation of system
safety requirements. Our repair strategy achieves its goal by (1) localizing
faults spanning several lines of code, (2) simultaneously
resolving multiple interaction failures caused by independent faults,
(3) scaling repair strategies from the unit-level to the system-level,
and (4) resolving failures based on their order of severity. We have
evaluated our approach using two industrial ADS containing four
features. Our results show that our repair strategy resolves the
undesired interaction failures in these two systems in less than 16h
and outperforms existing automated repair techniques
Digital Twins Are Not Monozygotic -- Cross-Replicating ADAS Testing in Two Industry-Grade Automotive Simulators
The increasing levels of software- and data-intensive driving automation call
for an evolution of automotive software testing. As a recommended practice of
the Verification and Validation (V&V) process of ISO/PAS 21448, a candidate
standard for safety of the intended functionality for road vehicles,
simulation-based testing has the potential to reduce both risks and costs.
There is a growing body of research on devising test automation techniques
using simulators for Advanced Driver-Assistance Systems (ADAS). However, how
similar are the results if the same test scenarios are executed in different
simulators? We conduct a replication study of applying a Search-Based Software
Testing (SBST) solution to a real-world ADAS (PeVi, a pedestrian vision
detection system) using two different commercial simulators, namely,
TASS/Siemens PreScan and ESI Pro-SiVIC. Based on a minimalistic scene, we
compare critical test scenarios generated using our SBST solution in these two
simulators. We show that SBST can be used to effectively and efficiently
generate critical test scenarios in both simulators, and the test results
obtained from the two simulators can reveal several weaknesses of the ADAS
under test. However, executing the same test scenarios in the two simulators
leads to notable differences in the details of the test outputs, in particular,
related to (1) safety violations revealed by tests, and (2) dynamics of cars
and pedestrians. Based on our findings, we recommend future V&V plans to
include multiple simulators to support robust simulation-based testing and to
base test objectives on measures that are less dependant on the internals of
the simulators.Comment: To appear in the Proc. of the IEEE International Conference on
Software Testing, Verification and Validation (ICST) 202
Test Generation and Test Prioritization for Simulink Models with Dynamic Behavior
All engineering disciplines are founded and rely on models, although they may differ on purposes and usages of modeling. Among the different disciplines, the engineering of Cyber Physical Systems (CPSs) particularly relies on models with dynamic behaviors (i.e., models that exhibit time-varying changes). The Simulink modeling platform greatly appeals to CPS engineers since it captures dynamic behavior models. It further provides seamless support for two indispensable engineering activities: (1) automated verification of abstract system models via model simulation, and (2) automated generation of system implementation via code generation.
We identify three main challenges in the verification and testing of Simulink models with dynamic behavior, namely incompatibility, oracle and scalability challenges. We propose a Simulink testing approach that attempts to address these challenges. Specifically, we propose a black-box test generation approach, implemented based on meta-heuristic search, that aims to maximize diversity in test output signals generated by Simulink models. We argue that in the CPS domain test oracles are likely to be manual and therefore the main cost driver of testing. In order to lower the cost of manual test oracles, we propose a test prioritization algorithm to automatically rank test cases generated by our test generation algorithm according to their likelihood to reveal a fault.
Engineers can then select, according to their test budget, a subset of the most highly ranked test cases. To demonstrate scalability, we evaluate our testing approach using industrial Simulink models. Our evaluation shows that our test generation and test prioritization approaches outperform baseline techniques that rely on random testing and structural coverage
Schedulability Analysis of Real-Time Systems with Uncertain Worst-Case Execution Times
Schedulability analysis is about determining whether a given set of real-time
software tasks are schedulable, i.e., whether task executions always complete
before their specified deadlines. It is an important activity at both early
design and late development stages of real-time systems. Schedulability
analysis requires as input the estimated worst-case execution times (WCET) for
software tasks. However, in practice, engineers often cannot provide precise
point WCET estimates and prefer to provide plausible WCET ranges. Given a set
of real-time tasks with such ranges, we provide an automated technique to
determine for what WCET values the system is likely to meet its deadlines, and
hence operate safely. Our approach combines a search algorithm for generating
worst-case scheduling scenarios with polynomial logistic regression for
inferring safe WCET ranges. We evaluated our approach by applying it to a
satellite on-board system. Our approach efficiently and accurately estimates
safe WCET ranges within which deadlines are likely to be satisfied with high
confidence
- …